Skip to content

Extract code generation logic into internal/api package#4418

Open
kyleconroy wants to merge 8 commits intomainfrom
claude/create-internal-api-package-INVIA
Open

Extract code generation logic into internal/api package#4418
kyleconroy wants to merge 8 commits intomainfrom
claude/create-internal-api-package-INVIA

Conversation

@kyleconroy
Copy link
Copy Markdown
Collaborator

@kyleconroy kyleconroy commented Apr 28, 2026

Introduce internal/api, a small programmatic interface to sqlc's code generation, inspired by esbuild's Build API. One entry point, api.Generate(ctx, api.GenerateOptions{}), returns a GenerateResult. The CLI becomes a thin wrapper.

Public surface

The package exports exactly three names:

func Generate(ctx context.Context, opts GenerateOptions) GenerateResult

type GenerateOptions struct {
    Config               io.Reader // sqlc YAML/JSON config; required
    Stderr               io.Writer
    Write                bool      // write generated files to disk
    Diff                 bool      // diff generated files against on-disk
    BaseDir              string    // resolves relative paths in Config; strips prefix in stderr labels
    EnableProcessPlugins bool      // permit process-based plugins to run
}

type GenerateResult struct {
    Files  map[string]string
    Errors []error
}

Everything else in the package — parse, codegen, the plugin shim, the result processor — is unexported.

Notable design choices

  • Config as io.Reader. No Dir/File fields. Callers hand sqlc whatever bytes they want. The CLI reads from disk; library callers can construct configs in memory.
  • BaseDir does double duty. It's the directory relative paths in the config resolve against and the prefix stripped from file paths in parse errors and diff labels. When empty it defaults to the current working directory. Six fields total, no separate "dir for resolving" vs. "dir for labels."
  • Process plugins are off by default. EnableProcessPlugins is a single bool whose zero value refuses any process plugin in the config — process plugins execute arbitrary local commands, so callers must opt in. The CLI sets it from env.Debug.ProcessPlugins, which defaults to true and flips off under SQLCDEBUG=processplugins=0.
  • Write/Diff replace separate functions. sqlc generate is api.Generate{Write: true}, sqlc compile is neither, sqlc diff is {Diff: true}. cmd.Diff is gone.

CLI

internal/cmd/cmd.go's genCmd, checkCmd, and diffCmd each:

  1. Read the config bytes.
  2. os.Chdir into the config's directory so relative paths resolve.
  3. Call api.Generate with Config: bytes.NewReader(data), BaseDir: configDir, and EnableProcessPlugins: env.Debug.ProcessPlugins.

cmd.Vet and cmd.Push still use the cmd-local helpers (parse, processQuerySets, codeGenRequest) since they have surface beyond what api covers; both packages skip joining their dir parameter when the path is already absolute, so configs with absolute paths flow through both.

Endtoend tests

internal/endtoend/endtoend_test.go calls api.Generate directly. A small mutatedConfigBytes helper parses the test's config, optionally applies a mutation (the managed-db context adds servers + sets database.managed), forces version "2" so v1 configs round-trip cleanly, and re-encodes as YAML. When mutated, the bytes are also dropped to a temp file alongside the original so cmd.Vet (which still takes a path) can use it.

Per-test environment variables from exec.json are applied via t.Setenv, and cmd.Env is then populated via opts.DebugFromEnv/ExperimentFromEnv — same path the CLI takes.

config.AnalyzerDatabase gained MarshalYAML/MarshalJSON so the parsed Config round-trips through yaml.Marshal cleanly — needed for the test helper.

Files

  • New: internal/api/{api,generate,process,parse,codegen,shim,diff}.go
  • Modified: internal/cmd/{cmd,generate,vet,process}.go, internal/config/config.go, internal/endtoend/endtoend_test.go, internal/endtoend/vet_test.go
  • Removed: internal/cmd/diff.go

Test plan

  • go test ./internal/endtoend/... (TestExamples, TestReplay base + managed-db) passes locally
  • go test ./... passes outside of pre-existing MySQL infra failures (TestExpandMySQL, TestValidSchema/*/mysql/*, TestExamplesVet/{authors,booktest,ondeck})
  • go build ./... and go vet ./... clean

claude added 5 commits April 28, 2026 05:33
The new package mirrors esbuild's Build API: a single api.Generate(ctx,
api.GenerateOptions{}) call returns a GenerateResult containing the
generated files and any errors. Most of cmd/generate.go's logic moves
here as unexported helpers; the only exported names are Generate,
GenerateOptions, and GenerateResult.

cmd.Generate is now a thin wrapper that translates the CLI's Options
struct into api.GenerateOptions. The endtoend tests call api.Generate
directly for TestExamples, TestReplay (generate command), and the
benchmarks.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
The two new boolean options let api.Generate cover the writefiles loop
and the diff comparison that previously lived in cmd. The compile
command becomes Generate with neither flag set, generate maps to
Write: true, and diff maps to Diff: true.

While simplifying GenerateOptions:
* Drop MutateConfig — tests now express config mutations by writing a
  temporary configuration file via writeMutatedConfig and pointing
  GenerateOptions.File at it. The mutated config is parsed (always to
  v2 shape), forced to version "2", and round-tripped via yaml.
* Drop DisableProcessPlugins from the API surface; we will revisit how
  to express that constraint.
* Add MarshalJSON/MarshalYAML to AnalyzerDatabase so the parsed Config
  round-trips through yaml.Marshal cleanly, which is what the new test
  helper relies on.

cmd/diff.go is gone and cmd/generate.go is left with only the helpers
(readConfig, parse, printFileErr) other cmd commands still use.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
Add an explicit allowlist of process-based plugin names to
api.GenerateOptions. Generate fails before any parse or codegen runs if
the configuration declares a process plugin whose name is not in the
list. The "Insecure" prefix mirrors crypto/tls.Config.InsecureSkipVerify
to flag the trust decision callers are making — process plugins execute
arbitrary local commands.

The CLI populates the allowlist by scanning the user's own config for
declared process plugins, so `sqlc generate`, `sqlc compile`, and
`sqlc diff` keep working. SQLCDEBUG=processplugins=0 still disables
process plugins by leaving the allowlist nil.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
The struct collapses to five fields: Config (io.Reader), Stderr, Write,
Diff, InsecureProcessPluginNames. api.Generate parses the config from
the reader and treats every relative path in it as relative to the
current working directory.

CLI: each command opens the config file, reads its bytes, parses it once
to extract declared process-plugin names, then chdirs to the config's
directory before invoking api.Generate. Single-process so chdir is fine.

Tests: a new mutatedConfigBytes helper parses the test's sqlc.yaml,
forces version "2", rewrites every schema/queries/output path to be
absolute relative to the test directory, and re-encodes as YAML — so
api.Generate works without knowing the source directory. Optional
mutate callback applies extra changes (managed-db servers etc.) and
also drops a temp file alongside the original for cmd.Vet which still
takes a config path.

cmd/process.go and cmd/vet.go now skip joining their dir parameter when
the config-supplied path is already absolute.

KNOWN ISSUE: TestReplay parse-error tests and the diff_output tests
fail because the api now emits absolute paths in error messages and
unified-diff labels (no config-dir context to strip). Either add a
BaseDir hint back to GenerateOptions or update the affected test
expectations to match.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
Restore the dir context that vanished with the io.Reader switch, but as
a single optional field. BaseDir is the directory relative paths in the
config (schema, queries, output) are resolved against, and the prefix
stripped from file paths shown in parse errors and diff labels. When
empty, BaseDir defaults to the current working directory.

A small resolvePath helper sits in api/generate.go and is called from
processQuerySets and ProcessResult; absolute paths pass through
unchanged. Parse errors and diff labels reuse the same BaseDir for
relative formatting.

Side-effects:
* The endtoend tests no longer pre-rewrite paths to absolute — they
  just pass BaseDir = test directory. The absolutizePaths helper is
  gone.
* The CLI sets BaseDir to the config's directory (returned by
  loadConfig) but keeps the chdir for now since other cmd paths
  (vet, push) still use cwd-relative resolution.

Tradeoff vs. the strictly-cosmetic option: GenerateOptions still has
six fields, but BaseDir's semantics are coherent with how it's used
(resolution + label) and library callers don't need to chdir or
hand-rewrite paths.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
@kyleconroy kyleconroy marked this pull request as ready for review May 2, 2026 04:13
claude added 3 commits May 2, 2026 04:24
A single bool is enough — the previous allowlist by-name was overkill
once we accepted the CLI is responsible for translating SQLCDEBUG into
the option. EnableProcessPlugins=false (zero value) refuses any process
plugin in the config; the CLI sets it to env.Debug.ProcessPlugins which
defaults to true and flips off under SQLCDEBUG=processplugins=0.

This drops the config pre-parse the CLI was doing to extract plugin
names; loadConfig now just reads bytes and chdirs.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
t.Setenv restores the previous value when the test exits, which avoids
leaking SQLC_DUMMY_VALUE and the per-example VET_TEST_* URIs into other
tests in the same process.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
Mirror how the real CLI populates cmd.Env: t.Setenv every entry from
exec.json's Env map and then call opts.DebugFromEnv/ExperimentFromEnv.
This generalises to any env var a test wants to set (not just
SQLCDEBUG/SQLCEXPERIMENT) and keeps the test path closer to the
production code path.

https://claude.ai/code/session_01RCzB2JR5Y5ScFDUmwcxGVZ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants